This PR introduces the FileService as another microservice parallel to WorkflowCompilingService, ComputingUnitMaster/Worker, and TexeraWebApplication.

Purpose of the FileService

• We want to improve the performance of our current Git-based Dataset implementation;
• We decide to go with LakeFS + S3, LakeFS for the version control metadata and S3 for data transfer; But LakeFS doesn't have access control layer
• Therefore, we build the FileService, providing
  • all the APIs related to versioned files in datasets
  • access control

Architecture before and after adding FileService

Before:

After

Key Changes

• A new service, FileService is introduced. All the dataset-related endpoints are hosted on FileService
• Several configuration items related to LakeFS and S3 are introduced in the storage-config.yaml
• Frontend UI updates to incorporate with new changes
• For ComputingUnitMaster and ComputingUnitWorker, they will call FileService to read files, during which their access will be verified. So in the dynamic computing architecture (which will be introduced in Add computing unit manager service #3298), they will send requests along with current user's token. In single-machine architecture, they are bypassing the network requests by doing direct local function calls.
• Python UDF can now directly read dataset's file by the following example code:

file = DatasetFileDocument("The URL of the file")
bytes = file.read_file() # return an io.Bytes object

You may refer to core/amber/src/main/python/pytexera/storage/dataset_file_document.py for implementation details. This feature is only available in the dynamic computing architecture.

How to migrate the previous datasets to the new datasets managed by the LakeFS

As we did quite some refactoring, two dataset implementations are NOT compatible with each others. To migrate the previous datasets to the latest implementation, you will need to re-upload the data via the new UI.

How to deploy new architecture

Step1. Deploy LakeFS & Minio

Use Docker (Highly recommended for local development)

• Go to directory: core/file-service/src/main/resources
• Execute docker-compose --profile local-lakefs up -d at its directory

Use Binary (Recommended for production deployment)

Refer to https://docs.lakefs.io/howto/deploy/

Step2. Configure the storage-config.yaml

Configure the below section in the storage-config.yaml:

  lakefs:
    endpoint: ""
    auth:
      api-secret: ""
      username: ""
      password: ""
    block-storage:
      type: ""
      bucket-name: ""

  s3:
    endpoint: ""
    auth:
      username: ""
      password: ""

Here is the configuration you can directly use if you are using the core/file-service/src/main/resources/docker-compose.yml to install LakeFS & Minio:

  lakefs:
    endpoint: "http://127.0.0.1:8000/api/v1"
    auth:
      api-secret: "random_string_for_lakefs"
      username: "AKIAIOSFOLKFSSAMPLES"
      password: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    block-storage:
      type: "s3"
      bucket-name: "texera-dataset"

  s3:
    endpoint: "http://localhost:9000"
    auth:
      username: "texera_minio"
      password: "password"

Step3. Launch services

Launch FileService , in addition to TexeraWebApplication, WorkflowCompilingService and ComputingUnitMaster.

Future PRs after this one

• Remove the Dataset related endpoints completely from amber package.
• Incorporate the deployment of LakeFS+S3 in the Helm chart of K8s-based deployment.
• Some optimizations:
  • for small files, directly upload it instead of using multipart upload
  • when doing result exports, use multipart upload as the result size can be quite big.
  • support re-transmit for partially-uploaded files.